Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints and Cognitive Glue

The Taxiway Navigation and Situation Awareness (T-NASA) system is a prototype augmented reality commercial airline cockpit display suite developed to increase efficiency and enhance situation awareness (SA) during airport surface taxi operations. The T-NASA system consists of a head-up display (HUD) and an electronic moving map (EMM), which allow for the display of navigation information using augmented reality techniques. The T-NASA HUD, an egocentric augmented reality display, uses "scene-linked symbology" (Foyle et al., 1996), derived from a set of theoretically-derived and experimentally-validated principles to enhance local guidance and route awareness. The EMM, is an exocentric perspective display that enhances route and global awareness. The principles for enhancing situation awareness in augmented reality systems are reviewed. 1 Problem: Airport Surface Operations The past few years in aviation have seen incredible technological advances in computing power and techniques, positioning and surveillance (e.g., Global Positioning System, GPS; ADS-B), avionics capabilities (e.g., head-up displays, HUDs; and, electronic moving maps, EMMs) and advances in display representations (e.g., "highway in the sky") that are now beginning to emerge into general aviation (GA) and commercial transport airline flight decks. Yet, many facets of our national airspace system (NAS) make little use of these technologies and suffer from issues of safety, efficiency and capacity. In its role of researching and developing advanced aviation systems to improve the efficiency and safety of the national airspace, one problem that NASA has addressed is the domain of airport surface (taxi) operations (Foyle, Andre, McCann, Wenzel, Begault & Battiste, 1996). This research and development was driven by three facts: (1) pilots have some difficulty taxiing, especially at night, or in lowvisibility, or at large, unfamiliar airports (Andre, 1995); (2) airplane taxi operations are the least technologicallyadvanced flight phase (Kelley & Adam, 1997); and, (3) safety improvements for surface operations are needed (FAA, 2002). In regards to technology, as Foyle et al. (1996) describes, from a pilot's perspective, taxiing between the runway and gate is done in the same manner that it has been done since the early days of passenger travel in the 1940's. The pilot contacts air traffic control (ATC) on a two-way radio, receives a verbal taxi route clearance, and then is required to follow that clearance on the airport surface. The only "tools" that the pilot has available, currently, are the airport signage denoting the taxi and runway names, and a standardized airport chart (paper hard-copy map) for reference. In regards to safety, the worst accident in aviation history, with 583 deaths, occurred on the airport surface at Tenerife airport in the Canary Islands, Spain in 1977. Recently, the FAA has targeted runway incursions (occurring when an airplane taxis onto an active runway without permission) as one of the FAA's highest priority safety issues (FAA, 2002). Clearly, many technological advances have occurred since 1940, and, in fact, these technologies are now finding their way through FAA certification into GA and airline flight decks. These new technologies pose both a challenge and an opportunity for designers of these emerging systems. 1.1 Display Information Requirements In many augmented reality applications, the transparent media device is a helmet-mounted display (HMD), that allows for the operator to "look around" in the world. In aviation, however, no HMD has been FAA-certified for civil aviation, although HMDs are in use in military rotorcraft. HUDs, are a fixed-forward, permanently-mounted, Foyle, D. C., Andre, A. D., & Hooey, B. L. (2005). Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints and Cognitive Glue. In Proceedings of the 11th International Conference on Human Computer Interaction. Las Vegas, NV. transparent display medium, are installed in many currently-operating airline flight decks, and have been since the early 1990s. To date, however, in airline operation, these HUDs are used primarily for take-off, approach, and landing, since the current display symbology is of limited value during other flight phases such as en route, or surface operations. In order to develop a system to address the problems outlined above associated with airport surface operations, an important phase of the design process was to define the precise nature of information required by the pilots, with particular emphasis on determining the information that pilots currently have available in clear visibility but is commonly degraded in low visibility (Hooey, Foyle & Andre; 2001). Lasswell and Wickens (1995) identified two classes of information necessary for successful taxi navigation, local guidance and global awareness, and expanded the subtasks inherent in each. Additionally, when ATC issues an assigned taxi clearance, a third piece of information is required, route awareness (Hooey, Foyle & Andre, 2001). The local guidance task of maneuvering the aircraft is comprised of lateral loop closure (e.g., minimizing lateral deviations), directional loop closure (e.g., steering around turns), longitudinal loop closure (e.g., maintaining speed and braking), hazard detection (e.g., monitoring for traffic and obstacles), and information gathering (e.g., scanning signage and markings). Pilots must also maintain a sense of global awareness, which refers to knowledge of the general layout of the airport surface, the location of their destination concourse or runway, and traffic. This information is necessary to navigate to a known destination, to avoid potential hazards along the way, and to recover quickly in the event of a navigation error. Lacking any one or more of these pieces of information can cause the pilot to become spatially disoriented. The results can range from an increase in workload to catastrophic accidents. In order to maneuver the aircraft on the correct route, pilots also need route awareness, or knowledge of their position on the airport surface relative to their cleared route. This includes knowing the name of the next required taxiway, the distance to the turn, and the direction of the turn. Thus, the requirements for the cockpit display system were to augment the local guidance information that is degraded in low visibility, replace the global awareness information that is missing in low visibility, and provide route awareness information that is currently lacking even in clear visibility. 2 Display Representations in Aviation In designing a display suite to provide the diverse types of information (local guidance, route awareness, and global awareness as described above), it was readily apparent that no single display could effectively portray all types of information. The design decision to allocate information across two different displays was driven by research on aviation display viewpoint placement (as summarized recently by Wickens, 2002, but also see Wickens & Prevett, 1995). Wickens characterizes the relation among the various viewpoints and has summarized the literature, display implications, as well as the advantages and disadvantages of each for aviation applications. Wickens notes that there are 4 cardinal eyepoints that can be described. Three of these are described sequentially, referring to Figure 1, beginning at the bottom and moving clockwise. First is the egocentric viewpoint, the nominal augmented-reality viewpoint. Here, the viewpoint is co-located with the pilot operator's real viewpoint, the eyes. The egocentric display shown is that of a "highway/tunnel in the sky" (Beringer, 2000), in which a conformal, virtual "tunnel" is shown on the display to allow the user to see an overlaid projection of the flight path. Next is an exocentric perspective representation. It is exocentric in that the viewpoint is no longer co-located with the operator/pilot, but is now located above and behind the aircraft with a depression angle of about 45 deg commonly used. The representative display shown allows the user to see the "ownship," the aircraft containing the user/pilot, as well as to see how one is referenced to upcoming objects in the world in front of the ownship. The final viewpoint shown in Figure 1 (top) is the exocentric 2-D (two-dimensional) plan view, which is a degenerative condition where the viewpoint is 90 deg to the world's plane, looking straight down in plan view. This gives rise to a 2-D representation shown in the corresponding representative display. (The fourth cardinal eyepoint described by Wickens is the exocentric 2-D side view, which is not shown here for simplicity.) At the highest level of abstraction, Wickens makes the point that as one progresses from egocentric through exocentric perspective to 2-D exocentric viewpoints, the displays associated with them become progressively less egocentric, less integrated, and are poorer ecological representations of the pilot's relation to the world. As the display viewpoint changes in the manner shown in Figure 1, in order, from egocentric to exocentric perspective, and then to 2-D plan-view, three display format implications are determined. First, the most obvious change is that the objects in the world are represented differently on the display, consistent with the optical rules of 2-D projection (e.g., interposition, perspective, etc.). This can be seen in Figure 1 where the mountains change from a side view to a top-down view as the display changes from egocentric to plan view. Second, the display Foyle, D. C., Andre, A. D., & Hooey, B. L. (2005). Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints and Cognitive Glue. In Proceedings of the 11th International Conference on Human Computer Interaction. Las Vegas, NV. augmentations available to display designers are changed. For example, the representation of a "tunnel in the sky" flight path indicator changes from that of an immersive tunnel path, to a less useful tunnel viewed from the outside, and then to a 2-D path, as the viewpoint changes from egocentric to exocentric perspective to 2-D plan-view. Finally, as mentioned previously, the displays are progressively less egocentric, less integrated, and are poorer ecological representations of the pilot's relation to the world. Depending on one's display design philosophy and system requirements, an immersive, integrated, ecologically-natural display representation may be desired.